Systems and methods for improving video captured using mobile devices

ABSTRACT

Systems, methods, and computer program products for capturing and analyzing image data, preferably video data, are disclosed. The inventive concepts include using multiple frames of image data to generate a composite image, where the composite image may be characterized by a higher resolution than one or more of the individual frames used to generate the composite image, and/or absence of a blurred region present in one or more of the individual frames. Inventive techniques also include determining a minimum capture resolution appropriate for capturing images of particular objects for downstream processing, and optionally triggering generation of a composite image having sufficient resolution to facilitate the downstream processing in response to detecting one or more frames of image data are characterized by a resolution, and/or a region having a resolution, less than the minimum capture resolution appropriate for capturing images of those particular objects.

PRIORITY CLAIM

This application is a continuation of U.S. patent application Ser. No.14/473,950, filed Aug. 29, 2014, which is a continuation of U.S. patentapplication Ser. No. 14/268,876, filed May 2, 2014, now U.S. Pat. No.8,885,229, which claims priority to U.S. Provisional Patent ApplicationNo. 61/819,463, filed May 3, 2013, each of which are herein incorporatedby reference.

RELATED APPLICATIONS

This application is related to U.S. Pat. No. 6,370,277, granted Apr. 9,2002 (U.S. patent application Ser. No. 09/206,753, filed Dec. 7, 1998)as well as copending U.S. patent application Ser. No. 13/740,123, filedJan. 11, 2013; Ser. No. 13/802,226, filed Mar. 13, 2013; Ser. No.14/209,825, filed Mar. 13, 2014; Ser. No. 14/259,866, filed Apr. 23,2014; Ser. No. 14/177,136, filed Feb. 10, 2014; Ser. No. 14/175,999,filed Feb. 7, 2014; Ser. No. 14/220,016, filed Mar. 19, 2014; Ser. No.14/220,023, filed Mar. 19, 2014 and Ser. No. 14/220,029, filed Mar. 19,2014; and Provisional U.S. Patent Application No. 61/883,865, filed Sep.27, 2013, and 61/905,063, filed Nov. 15, 2013, each of which are hereinincorporated by reference.

FIELD OF INVENTION

The present invention relates to digital video capture and digital videodata processing, and more particularly to capturing and processingdigital video data using a mobile device.

BACKGROUND OF THE INVENTION

Modern mobile devices are well adapted to capturing images of a varietyof objects, including documents, persons, automobiles, etc. Improvementsto the mobile device camera capabilities and/or processing power makeapplications for capturing and/or processing digital image data using amobile device increasingly attractive in an increasinglymobile-device-driven economy.

However, limitations of the mobile device hardware and practicallimitations of capturing images using a mobile device present majorchallenges to efficient and effective digital image processing. Forexample, digital images captured using a mobile device are often ofinsufficient quality for subsequent processing due to one or moreartifacts such as blur, uneven illumination, insufficient illumination,oversaturated illumination, insufficient resolution, projective effects,etc. Attempts to process digital images including such artifacts mayfail completely or produce inadequate quality results for the desiredapplication. At best, the user may be required to repeat the captureoperation and attempt to improve the quality of the image, but in somecases recapturing the image may be impossible, resulting in lostopportunity for acquiring images of important but transientcircumstances, such as the location or condition of a person or vehiclebefore, during, and/or after an automobile accident.

Accordingly, it would be beneficial to provide systems, methods, and/orcomputer program products capable of capturing and/or processing dataother than still digital images in a manner that overcomes thechallenges presented above and improve users' ability to capture andprocess data, especially using mobile devices.

SUMMARY OF THE INVENTION

In one embodiment, a method includes: invoking an image captureinterface via a mobile device, the capture interface comprising aviewfinder represented on a display of the mobile device; analyzing aplurality of frames of video data captured via the capture interface,wherein the analyzing comprises determining: whether an objectexhibiting one or more defining characteristics is depicted within theviewfinder; and whether the object depicted within the viewfindersatisfies one or more predetermined quality control criteria; and inresponse to determining a frame fails one or more of the predeterminedquality control criteria, displaying an indication of the failure on themobile device display; and in response to determining the objectdepicted within the viewfinder satisfies the one or more predeterminedquality control criteria, one or more of: displaying an indication thatthe object depicted in the viewfinder exhibits the one or more definingcharacteristics; automatically capturing an image of the object, whereinthe image is characterized by a resolution higher than a resolution ofthe video data; and automatically storing to a memory one or more of theframes in which the object satisfying the predetermined quality controlcriteria is depicted in the viewfinder.

In another embodiment, a system includes: a processor; and logic inand/or executable by the processor to cause the processor to: invoke animage capture interface via a mobile device, the capture interfacecomprising a viewfinder represented on a display of the mobile device;analyze a plurality of frames of video data captured via the captureinterface, wherein the analysis comprises: determining whether an objectexhibiting one or more defining characteristics is depicted within theviewfinder; and determining whether the object depicted within theviewfinder satisfies one or more predetermined quality control criteria;and in response to determining a frame fails one or more of thepredetermined quality control criteria, display an indication of thefailure on the mobile device display; and in response to determining theobject depicted within the viewfinder satisfies the one or morepredetermined quality control criteria, one or more of: display anindication that the object depicted in the viewfinder exhibits the oneor more defining characteristics; automatically capture an image of theobject, wherein the image is characterized by a resolution higher than aresolution of the video data; and automatically store to a memory one ormore of the frames in which the object satisfying the predeterminedquality control criteria is depicted in the viewfinder.

In still another embodiment, a computer program product includes: acomputer readable storage medium having program code embodied therewith,the program code readable/executable by a processor to cause theprocessor to: invoke an image capture interface via a mobile device, thecapture interface comprising a viewfinder represented on a display ofthe mobile device; analyze a plurality of frames of video data capturedvia the capture interface, wherein the analysis comprises: determiningwhether an object exhibiting one or more defining characteristics isdepicted within the viewfinder; and determining whether the objectdepicted within the viewfinder satisfies one or more predeterminedquality control criteria; and in response to determining a frame failsone or more of the predetermined quality control criteria, display anindication of the failure on the mobile device display; and, in responseto determining the object depicted within the viewfinder satisfies theone or more predetermined quality control criteria, one or more of:display an indication that the object depicted in the viewfinderexhibits the one or more defining characteristics; automatically capturean image of the object, wherein the image is characterized by aresolution higher than a resolution of the video data; and automaticallystore to a memory one or more of the frames in which the objectsatisfying the predetermined quality control criteria is depicted in theviewfinder.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a network architecture, in accordance with oneembodiment.

FIG. 2 shows a representative hardware environment that may beassociated with the servers and/or clients of FIG. 1, in accordance withone embodiment.

FIGS. 3-5 each depict a flowchart of a method, according to oneembodiment.

DETAILED DESCRIPTION

The following description is made for the purpose of illustrating thegeneral principles of the present invention and is not meant to limitthe inventive concepts claimed herein. Further, particular featuresdescribed herein can be used in combination with other describedfeatures in each of the various possible combinations and permutations.

Unless otherwise specifically defined herein, all terms are to be giventheir broadest possible interpretation including meanings implied fromthe specification as well as meanings understood by those skilled in theart and/or as defined in dictionaries, treatises, etc.

It must also be noted that, as used in the specification and theappended claims, the singular forms “a,” “an” and “the” include pluralreferents unless otherwise specified.

The present application refers to image processing. In particular, thepresent application discloses systems, methods, and computer programproducts designed to augment current still-photo based methods andsystems for capturing a digital image to leverage streaming video datato capture and process various types of information. As digital imagingtechnology continues to improve, video stream resolution usually lagsavailable photo resolutions. As such, leveraging video streaming fordocument capture has been previously limited to smaller size objectsbecause the available resolution did not always provide sufficientdetail regarding larger objects to effectively process digital imagesdepicting such objects. As video stream resolutions increase, thislimitation has decreasingly impacted processing capabilities andsuitability such that processing data from a video stream has become aviable alternative or even a preferred replacement to still-photocapture in various applications and/or implementations.

At a high level, an exemplary user experience for video stream capturemay be loosely based on the following scenario.

A user invokes a video-stream capture interface from a native mobileapplication, via a software development kit (SDK) used to develop ormodify anew or existing mobile application, via a built-in mobileoperating system (OS) functionality, etc. Once invoked, the user ispresented with an option to select video-based capture and perform avideo-based capture operation. The capture application exposes a videocapture interface that guides the user to ensure the physical objectremains within the bounds of a bounding box superimposed on the mobilecapture user interface. Once within the bounding box, the user clicks onthe ‘Capture’ button to initiate the capture process. Once initiated, amobile interface begins checking for stability of the mobile deviceusing one or more hardware components of the mobile device such as anaccelerometer, gyroscope, etc. Once stability has been achieved, anautofocus operation may be forced and the process of analyzing each ofthe (n) frames of the video stream begins.

The goal of frame analysis is to detect the existence of a target objectwithin the vantage point provided by the video stream. Entities includebut are not limited to page(s), barcode(s), buildings, motor vehicles,boats, persons, etc. The actual implementation of the real-timemethodology and algorithms used to detect the existence of the targetentity within the video frame will be discussed separately from thisdocument.

Once the existence of the target object has been detected in one or moreframes of the stream, either the frame is identified and processed byimage perfection techniques, such as embodied in one exemplary scenariovia electronic virtual rescan (EVRS) or for devices that support thenecessary capability, the full resolution (photo) corresponding to thetarget video frame is identified and processed by EVRS. Alternatively,multiple low-resolution video frames could be combined to a singlehigher-resolution image.

From there, the mobile application may facilitate providing as muchrelevant entity metadata as possible with the lowest possible latency.Relevant metadata could include but not be limited to object type,object characteristics, field metadata. GPS information, page size,barcode value(s), car type, person height, boat length, etc.).

This capability would allow the user to capture multiple objects andobject types simultaneously. Moreover, objects may be associated with aparticular downstream process (e.g. a business process such as a loanapplication, insurance claim, financial transaction, etc.) quickly andeasily with minimal user input other than simple click, point andcapture functionality.

From one perspective, the overall capture and processing may generallyfollow a logical order similar to the flow diagram shown below.

It will be appreciated upon reading the present descriptions that theoverall flow diagram shown below is a coarse conceptual example thatshould not be considered limiting in any way. The presently describedcapture and processing may, in various embodiments, include any numberof additional and/or different operations, perform such operations in adifferent order, and/or omit certain operations depicted in the flowdiagram.

Images (e.g. pictures, figures, graphical schematics, single frames ofmovies, videos, films, clips, etc.) are preferably digital imagescaptured by cameras, especially cameras of mobile devices. As understoodherein, a mobile device is any device capable of receiving data withouthaving power supplied via a physical connection (e.g. wire, cord, cable,etc.) and capable of receiving data without a physical data connectione.g. wire, cord, cable, etc.). Mobile devices within the scope of thepresent disclosures include exemplary devices such as a mobiletelephone, smartphone, tablet, personal digital assistant, iPod®, iPad®,BLACKBERRY® device, etc.

However, as it will become apparent from the descriptions of variousfunctionalities, the presently disclosed mobile image processingalgorithms can be applied, sometimes with certain modifications, toimages coining from scanners and multifunction peripherals (MFPs).Similarly, images processed using the presently disclosed processingalgorithms may be further processed using conventional scannerprocessing algorithms, in some approaches.

Of course, the various embodiments set forth herein may be implementedutilizing hardware, software, or any desired combination thereof. Forthat matter, any type of logic may be utilized which is capable ofimplementing the various functionality set forth herein.

One benefit of using a mobile device is that with a data plan, imageprocessing and information processing based on captured images can bedone in a much more convenient, streamlined and integrated way thanprevious methods that relied on presence of a scanner. However, the useof mobile devices as document(s) capture and/or processing devices hasheretofore been considered unfeasible for a variety of reasons.

In one approach, an image may be captured by a camera of a mobiledevice. The term “camera” should be broadly interpreted to include anytype of device capable of capturing an image of a physical objectexternal to the device, such as a piece of paper. The term “camera” doesnot encompass a peripheral scanner or multifunction device. Any type ofcamera may be used. Preferred embodiments may use cameras having ahigher resolution, e.g. 8 MP or more, ideally 12 MP or more. The imagemay be captured in color, grayscale, black and white, or with any otherknown optical effect. The term “image” as referred to herein is meant toencompass any type of data corresponding to the output of the camera,including raw data, processed data, etc.

The description herein is presented to enable any person skilled in theart to make and use the invention and is provided in the context ofparticular applications of the invention and their requirements. Variousmodifications to the disclosed embodiments will be readily apparent tothose skilled in the art and the general principles defined herein maybe applied to other embodiments and applications without departing fromthe spirit and scope of the present invention. Thus, the presentinvention is not intended to be limited to the embodiments shown, but isto be accorded the widest scope consistent with the principles andfeatures disclosed herein.

In particular, various embodiments of the invention discussed herein areimplemented using the Internet as a means of communicating among aplurality of computer systems. One skilled in the art will recognizethat the present invention is not limited to the use of the Internet asa communication medium and that alternative methods of the invention mayaccommodate the use of a private intranet, a Local Area Network (LAN), aWide Area Network (WAN) or other means of communication. In addition,various combinations of wired, wireless e.g., radio frequency) andoptical communication links may be utilized.

The program environment in which one embodiment of the invention may beexecuted illustratively incorporates one or more general-purposecomputers or special-purpose devices such hand-held computers. Detailsof such devices (e.g., processor, memory, data storage, input and outputdevices) are well known and are omitted for the sake of clarity.

It should also be understood that the techniques of the presentinvention might be implemented using a variety of technologies. Forexample, the methods described herein may be implemented in softwarerunning on a computer system, or implemented in hardware utilizing oneor more processors and logic (hardware and/or software) for performingoperations of the method, application specific integrated circuits,programmable logic devices such as Field Programmable Gate Arrays(FPGAs), and/or various combinations thereof. In one illustrativeapproach, methods described herein may be implemented by a series ofcomputer-executable instructions residing on a storage medium such as aphysical (e.g., non-transitory) computer-readable medium. In addition,although specific embodiments of the invention may employobject-oriented software programming concepts, the invention is not solimited and is easily adapted to employ other forms of directing theoperation of a computer.

The invention can also be provided in the form of a computer programproduct comprising a computer readable storage or signal medium havingcomputer code thereon, which may be executed by a computing device(e.g., a processor) and/or system. A computer readable storage mediumcan include any medium capable of storing computer code thereon for useby a computing device or system, including optical media such as readonly and writeable CD and DVD, magnetic memory or medium (e.g., harddisk drive, tape), semiconductor memory (e.g., FLASH memory and otherportable memory cards, etc.), firmware encoded in a chip, etc.

A computer readable signal medium is one that does not fit within theaforementioned storage medium class. For example, illustrative computerreadable signal media communicate or otherwise transfer transitorysignals within a system, between systems e.g., via a physical or virtualnetwork, etc.

FIG. 1 illustrates an architecture 100, in accordance with oneembodiment. As shown in FIG. 1, a plurality of remote networks 102 areprovided including a first remote network 104 and a second remotenetwork 106. A gateway 101 may be coupled between the remote networks102 and a proximate network 108. In the context of the present networkarchitecture 100, the networks 104, 106 may each take any formincluding, but not limited to a LAN, a WAN such as the Internet, publicswitched telephone network (PSTN), internal telephone network, etc.

In use, the gateway 101 serves as an entrance point from the remotenetworks 102 to the proximate network 108. As such, the gateway 101 mayfunction as a router, which is capable of directing a given packet ofdata that arrives at the gateway 101, and a switch, which furnishes theactual path in and out of the gateway 101 for a given packet.

Further included is at least one data server 114 coupled to theproximate network 108, and which is accessible from the remote networks102 via the gateway 101. It should be noted that the data server(s) 114may include any type of computing device/groupware. Coupled to each dataserver 114 is a plurality of user devices 116. Such user devices 116 mayinclude a desktop computer, laptop computer, hand-held computer, printeror any other type of logic. It should be noted that a user device 111may also be directly coupled to any of the networks, in one embodiment.

A peripheral 120 or series of peripherals 120, e.g. facsimile machines,printers, networked storage units, etc., may be coupled to one or moreof the networks 104, 106, 108. It should be noted that databases,servers, and/or additional components may be utilized with, orintegrated into, any type of network element coupled to the networks104, 106, 108. In the context of the present description, a networkelement may refer to any component of a network.

According to some approaches, methods and systems described herein maybe implemented with and/or on virtual systems and/or systems whichemulate one or more other systems, such as a UNIX system which emulatesa MAC OS environment, a UNIX system which virtually hosts a MICROSOFTWINDOWS environment, a MICROSOFT WINDOWS system which emulates a MAC OSenvironment, etc. This virtualization and/or emulation may be enhancedthrough the use of VMWARE software, in some embodiments.

In more approaches, one or more networks 104, 106, 1.08, may represent acluster of systems commonly referred to as a “cloud.” In cloudcomputing, shared resources, such as processing power, peripherals,software, data processing and/or storage, servers, etc., are provided toany system in the cloud, preferably in an on-demand relationship,thereby allowing access and distribution of services across manycomputing systems. Cloud computing typically involves an Internet orother high speed connection (e.g., 4G LTE, fiber optic, etc.) betweenthe systems operating in the cloud, but other techniques of connectingthe systems may also be used.

FIG. 1 illustrates an architecture 100, in accordance with oneembodiment. As shown in FIG. 1, a plurality of remote networks 102 areprovided including a first remote network 104 and a second remotenetwork 106. A gateway 101 may be coupled between the remote networks102 and a proximate network 108. In the context of the presentarchitecture 100, the networks 104, 106 may each take any formincluding, but not limited to a LAN, a WAN such as the Internet, publicswitched telephone network (PSTN), internal telephone network, etc.

In use, the gateway 101 serves as an entrance point from the remotenetworks 102 to the proximate network 108. As such, the gateway 101 mayfunction as a router, which is capable of directing a given packet ofdata that arrives at the gateway 101, and a switch, which furnishes theactual path in and out of the gateway 101 for a given packet.

Further included is at least one data server 114 coupled to theproximate network 108, and which is accessible from the remote networks102 via the gateway 101. It should be noted that the data server(s) 114may include any type of computing device/groupware. Coupled to each dataserver 114 is a plurality of user devices 116. Such user devices 116 mayinclude a desktop computer, lap-top computer, hand-held computer,printer or any other type of logic. It should be noted that a userdevice 111 may also be directly coupled to any of the networks, in oneembodiment.

A peripheral 120 or series of peripherals 120, e.g., facsimile machines,printers, networked and/or local storage units or systems, etc., may becoupled to one or more of the networks 104, 106, 108. It should be notedthat databases and/or additional components may be utilized with, orintegrated into, any type of network element coupled to the networks104, 106, 108. In the context of the present description, a networkelement may refer to any component of a network.

According to some approaches, methods and systems described herein maybe implemented with and/or on virtual systems and/or systems whichemulate one or more other systems, such as a UNIX system which emulatesa MAC OS environment, a UNIX system which virtually hosts a MICROSOFTWINDOWS environment, a MICROSOFT WINDOWS system which emulates a MAC OSenvironment, etc. This virtualization and/or mutation may be enhancedthrough the use of VMWARE software, in some embodiments.

In more approaches, one or more networks 104, 106, 108, may represent acluster of systems commonly referred to as a “cloud,” In cloudcomputing, shared resources, such as processing power, peripherals,software, data processing and/or storage, servers, etc., are provided toany system in the cloud, preferably in an on-demand relationship,thereby allowing access and distribution of services across manycomputing systems. Cloud computing typically involves an Internet orother high speed connection (e.g., 4G LTE, fiber optic, etc.) betweenthe systems operating in the cloud, but other techniques of connectingthe systems may also be used.

FIG. 2 shows a representative hardware environment associated with auser device 116 and/or server 114 of FIG. 1, in accordance with oneembodiment. Such figure illustrates a typical hardware configuration ofa workstation having a central processing unit 210, such as amicroprocessor, and a number of other units interconnected via a systembus 212.

The workstation shown in FIG. 2 includes a Random Access Memory (RAM)214, Read Only Memory (ROM) 216, an I/O adapter 218 for connectingperipheral devices such as disk storage units 220 to the bus 212, a userinterface adapter 222 for connecting a keyboard 224, a mouse 226, aspeaker 228, a microphone 232, and/or other user interface devices suchas a touch screen and a digital camera (not shown) to the bus 212,communication adapter 234 for connecting the workstation to acommunication network 235 e.g., a data processing network) and a displayadapter 236 for connecting the bus 212 to a display device 238.

The workstation may have resident thereon an operating system such asthe Microsoft Windows® Operating System (OS), a MAC OS, a UNIX OS, etc.It will be appreciated that a preferred embodiment may also beimplemented on platforms and operating systems other than thosementioned. A preferred embodiment may be written using JAVA, XML, C,and/or C++ language, or other programming languages, along with anobject oriented programming methodology. Object oriented programming(OOP), which has become increasingly used to develop complexapplications, may be used.

An application may be installed on the mobile device, e.g., stored in anonvolatile memory of the device. In one approach, the applicationincludes instructions to perform processing of an image on the mobiledevice. In another approach, the application includes instructions tosend the image to a remote server such as a network server. In yetanother approach, the application may include instructions to decidewhether to perform some or all processing on the mobile device and/orsend the image to the remote site.

In various embodiments, the presently disclosed methods, systems and/orcomputer program products may utilize and/or include any of thefunctionalities disclosed in related U.S. patent application Ser. No.13/740,123, filed Jan. 11, 2013. For example, digital images suitablefor processing in whole or in part using the presently disclosedmethodologies, systems, etc. may be subjected to any image processingoperations disclosed in the aforementioned Patent Application, such aspage detection, rectangularization, detection of uneven illumination,illumination normalization, resolution estimation, blur detection, etc.

In various embodiments, the presently disclosed methods, systems and/orcomputer program products may utilize and/or include any of thefunctionalities disclosed in related U.S. patent application Ser. No.13/802,226, filed Mar. 13, 2013 and Provisional U.S. Patent ApplicationNo. 61/780,747, filed Mar. 13, 2013. For example, digital imagessuitable for processing in whole or in part using the presentlydisclosed methodologies, systems, etc. may be subjected to anyclassification and/or data extraction operations disclosed in theaforementioned Patent Applications, including for instance classifyingobjects depicted in a digital image according to type based at least inpart on characteristics of the object, performing custom-tailored imageprocessing using information about the object characteristics and/orobject class, building and/or using feature vectors to performclassification, building and/or using feature vectors to develop a dataextraction model for the object and/or object class(es), using dataextraction models to extract data from digital images, etc.

In some embodiments, and entirely separate from the “extraction”operations and techniques disclosed herein, it may be useful to performthe image capture, analysis and processing as described, andsubsequently analyze the resulting image with a targeted opticalcharacter recognition (OCR) operation. For example, a user may define aportion of a processed image upon which to perform the OCR, and mayhover a window over that portion of the processed image. Then, the usermay receive OCR results either pursuant to a request submitted by theuser e.g. to OCR the windowed region of the image) or automatically innear- or real-time in response to the window position e.g. aconstantly-active OCR process is performed on the image portion(s)falling within the OCR window, and any recognized characters may bedisplayed in real-time on the mobile device.

Preferably, the windowed OCR approach may be utilized to determine,verify (e.g. confirm an observed value obtained via OCR by comparing toa reference value), and/or validate (e.g. as mentioned above and furtherdescribed in related U.S. Pat. No. 8,345,981 and/or U.S. patentapplication Ser. No. 14/175,999 (filed Feb. 7, 2014); Ser. No.14/176,606 (filed Feb. 7, 2014) and/or Ser. No. 14/078,402 (filed Nov.12, 2013)) text characters depicted in the depicted object. Even morepreferably, the windowed OCR approach may be utilized to specificallydetermine “identifying information,” e.g. as defined and described inrelated U.S. patent application Ser. No. 14/220,016 (filed Mar. 19,2014).

For example, in one approach classification may include determiningwhether a depicted object belongs to one or more predetermined classes,and if not, requesting user input defining a new class. This approachmay be augmented in some embodiments by automatically determiningdefining characteristics for the new class based on the user input, theobject depicted in the image(s), a combination thereof, and/or any otherrelevant descriptive information as would be appreciated by skilledartisans. In this manner, it is possible for the present systems to beextended to unknown object types based on minimal input from the userand defining characteristics determined based on user input, image data,and/or a combination thereof.

In more approaches, the presently disclosed methods, systems, and/orcomputer program products may be utilized with, implemented in, and/orinclude one or more user interfaces configured to facilitate performingany functionality disclosed herein and/or in the aforementioned relatedPatent Application, such as an image processing mobile application, acase management application, and/or a classification application, inmultiple embodiments.

In still more approaches, the presently disclosed systems, methodsand/or computer program products may be advantageously applied to one ormore of the use methodologies and/or scenarios disclosed in theaforementioned related Patent Application, among others that would beappreciated by one having ordinary skill in the art upon reading thesedescriptions.

It will further be appreciated that embodiments presented herein may beprovided in the form of a service deployed on behalf of a customer tooffer service on demand.

Video Capture and Discovery

In some embodiments, via a mobile application a user may capture video,analyze video and then store a full still photo resolution frame orframes. To facilitate computational efficiency, it is possible to usevideo data with a lower resolution than the full still photo resolutionframe(s) for discovering objects depicted in the frame(s). Upondiscovering a target object, various embodiments may use one or morehigh resolution photo frame for further processing.

For example, low-resolution video capture and processing of smalldocuments like drivers licenses or business cards or checks is possibleat least in part because some embodiments of capture may zoom in soclose that even the low resolution video feed produces sufficientresolution for discovering the small document in the object.

In one approach, a capture component of a mobile application within thescope of the present disclosure may facilitate a user invoking a mobiledevice camera in a video capture mode. The user may provide inputinstructing the capture component to initiate capturing video data. Theapplication, in response to receiving the “begin capture” instruction,in response to displaying a prompt to the user instructing the user toprepare for capturing data, etc. may query an on-device hardware such asan accelerometer and/or gyroscope for stability information. Upondetecting conditions from the on-device hardware that correspond to astability condition, the application may force an autofocus, captureframes, and then spawn a background process to invoke and/or conductimage processing.

Moreover, the captured frames may be characterized by a resolutionhigher than a resolution of the video stream (and corresponding data)displayed to the user while performing the stability determination,focus, object discovery, etc. In some embodiments, a user reviewing avideo stream may be simultaneously presented with a correspondinghigh-resolution frame of image data to review and/or provide feedbackand user input relating to capture and/or processing using the mobiledevice/application.

In more embodiments, the capture component may be further improved toclassify objects by type and selectively invoke the capture operation.For example, capture may be invoked only upon determining the capturefield encompasses an object of interest, such as a document, an animal,a vehicle, a person, a particular type of document, animal, vehicle,etc.

In still more embodiments, the capture component may be further improvedto determine classification of objects and/or detect characteristics ofobjects, and selectively invoke the capture operation in response todetecting an expected type of characteristic in the object. For example,a video stream of a capture field encompassing a document may beutilized to classify the type of document, and based on the documentclassification, the video stream may be utilized to determine whetherthe document contains particular characteristics, such as particularcontent (e.g. particular text such as a name, address, account number, aparticular symbol such as a barcode, logo, a photograph, etc. as wouldbe understood by one having ordinary skill in the art upon reading thepresent descriptions).

By providing additional classification capability in this vein, themobile application may avoid undesirably capturing video data uponreceiving information indicative of a stability condition, but where thecapture field is focused on an object of no interest for subsequentprocessing (e.g. the mobile application would be capable of selectivelyavoiding capturing video of a dog as part of an overalldocument-processing workflow). Preferably, the classifying operationcomprises a high-speed initial analysis to detect the existence of theobject of interest in the video frame. However, classification mayinclude any functionality discussed in related U.S. patent applicationSer. No. 13/802,226.

In various approaches, upon determining a stability exists, achievingfocus, and determining the capture field encompasses a desired capturetarget, the mobile application may invoke one or more processingoperations. As input to the processing operations, the mobileapplication may provide either the video stream, frames from the videostream, and/or high resolution equivalents thereof.

Additional embodiments may include providing functionality to determinea minimum resolution necessary to perform object discovery, imageprocessing, or any of a variety of downstream processing operations sothat a mobile application may facilitate a user capturing the requisitedata for subsequent processing in the most computationally efficientmanner possible. For example, a user may invoke a training component ofthe mobile application, and directed to capture video data correspondingto a particular object or object type; the user may be directed toperform the capture operation for a plurality of repetitions, and withor without the user's knowledge, each repetition may capture video dataat a different resolution so as to provide a diverse array of video datarepresenting the object in a range of resolutions.

The mobile application may transparently perform object discovery, imageprocessing, etc. using one or more of the plurality ofdifferent-resolution video data samples. Some of the samples may produceacceptable results, while others may not. The application may utilizeinformation regarding the results achieved using various input samplesto determine a resolution for subsequent use when capturing and/orperforming various processing operations for objects corresponding tothe particular object or object type for which training was performed.

In one approach, a resolution sufficient for object discovery is anyresolution that enables detection of contrast between the foreground ofthe image, e.g. regions of the image corresponding to the object, andthe background of the image, e.g. regions of the image not correspondingto the object. Detecting contrast includes detecting the existence of asubregion of the image containing a potential or “candidate” objectboundary. For certain objects, a resolution in a range from about 25dots per inch (DPI) to about 50 DPI may be sufficient to detect contrastand therefore object boundaries. Initial processing such as objectdiscovery may be performed using these relatively low-resolution imagesto process data in a highly efficient manner. Additional processing maybe performed utilizing the low-resolution image or a correspondinghigh-resolution image according to the requirements and/or desiredresult of the process.

In some approaches, upon detecting an object from the video stream data,a corresponding high resolution image may be captured and cropped toremove some or all background from the image.

In further embodiments, user feedback may be requested, obtained, and/orused to facilitate capturing and/or processing of video data asdescribed herein. For example, upon performing object discovery on videodata, various frames of the video data in which an object was reportedlydiscovered may be presented to the user. The user may confirm, modify ornegate the discovery result determination. Based on the user input, thediscovery algorithm may be modified. In another example, based on theuser input a minimum capture resolution may be determined, where thevarious frames correspond to different capture resolutions, as discussedabove in regard to training the capture component.

Super-Resolution

In further approaches, it may be advantageous to utilize data frommultiple frames of image and/or video data to generate a single,superior composite image for processing. For example, a higherresolution image may be composed from multiple relatively low-resolutionframes of video data. Alternatively, multiple high-resolution images maybe synthesized into an even higher-resolution image. Further still, arelatively low-resolution region of an otherwise high-resolution image,or a blurred region (for example as may be caused by unstable captureconditions) of an otherwise clear image may be improved by synthesizingdata from multiple image and/or video frames to resample thelow-resolution or blurred region and generate a high-quality (i.e. highresolution/clarity) composite image. In some embodiments, the frames mayrepresent binary image data (i.e. corresponding to two-tone or “bitonal”images), which may be compared, merged, and/or utilized to extract datafrom the image, such as text characters on a document.

Metadata

Retrieving, receiving, and providing metadata, as well as associatingmetadata with digital image data is another advantageous functionalitywithin the scope of the presently described mobile application.Preferably, the mobile application facilitates obtaining and associatingall available metadata with the corresponding image data. For example,in one scenario a user captures a video stream and/or image datacorresponding to a document. The document may be detected within theimage data and classified as a particular document type. Based on theclassification, metadata may be retrieved from a knowledge basecomprising a plurality of document classes and associated metadata. Theretrieved metadata may then be associated with the document image dataand/or video data in any suitable manner.

Metadata may include any information that is relevant to an object, animage of an object, etc. With continuing reference to the exemplaryscenario involving a document as the object, illustrative metadata mayinclude the document type, text content in the document, context of thetext (e.g. positional location, font type, color, size, etc.) page size,page resolution, color bit depth, etc. In other embodiments, themetadata may correspond to instructions for subsequent processing of thedata, such as particular parameters for manipulating image size, colorprofile, etc., particular parameters for extracting data from the image,etc. as would be understood by one having ordinary skill in the art uponreading the present descriptions.

Image Authentication

In some embodiments, various types of data, including raw and/orprocessed image data, metadata associated with image data, etc. as wouldbe understood by one having ordinary skill in the art reading thepresent descriptions can include and/or be associated withauthentication data. Authentication data may be utilized to very quicklyand efficiently determine a status of data, such as whether a particularfile has been altered from a previous state (e.g. by adding or removingmetadata to an image file, by editing metadata associated with an imagefile, by processing or re-processing image data with differentparameters, etc.).

In one embodiment, image authentication may include creating one or moreauthentication strings from a buffer in memory. The string may be anylength, but is preferably a 127-byte string in at least some approaches.More particularly, authentication may involve compressing an image(which may include image data and/or associated metadata) to a memorybuffer and creating the authentication string or strings for/from one ormore portions of the buffer. Creating the authentication string(s) mayinclude encoding data in the portions of the buffer into theauthentication string, in one embodiment. Any form of encoding may beemployed.

For example, authentication may generate an authentication string foronly the image, for only the metadata, for the image and the associatedmetadata, etc. The authentication strings may be inserted into, appendedto, or associated with the buffer in various approaches, and arepreferably added to the buffer as one or more tags, at which point thebuffer is dumped to physical memory (e.g. to disk) as a file. Notably,these authentication approaches may be equally applied to any format ofimage and/or metadata, including any compression format allowingmetadata tags, such as JPEG or TIFF formats.

Additionally and/or alternatively, the authentication string may beembedded into the image itself, for example using a steganographicapproach.

Data having authentication strings as described above can besubsequently authenticated to determine whether the data has beenaltered since the authentication strings were created. In particular,the file having the authentication tags may be read from physical memoryinto a memory buffer, and the encoded authentication strings may beextracted from the corresponding tags. These strings may be decoded andcompared to the corresponding portion(s) of the buffer from which theencoded authentication string was generated. If the decodedauthentication string and the portion of the buffer used to generate theauthentication string match, the portion of the buffer used to generatethe authentication string has not been altered, indicating that theentire file is unlikely to have been altered either. By utilizingmultiple authentication strings (multiple portions of the buffer),determining whether a file has been altered may be performed with higherconfidence, albeit at the cost of computational efficiency.

In one exemplary approach, video capture and processing may be performedin a manner substantially similar to the flow diagram shown below. Asnoted with regard to other flow diagrams presented above, thisillustrative example is in no way limiting, but rather provided tofacilitate better understanding of the inventive concepts presentedherein.

Video Capture User Interface

In still more embodiments, the presently described systems, methods, andcomputer program products may be implemented via one or more userinterfaces configured to facilitate capturing and processing informationusing video data.

The user interfaces may further enable a user to easily perform captureand processing operations using video data, as well as review theresults of such capture and/or processing operations in real-time ornear real-time. For example, each time that image and/or video data iscaptured and/or processed, a thumbnail corresponding to the image and/orvideo data may be produced and presented to a user. Generating thethumbnail may be a process that is performed asynchronously in thebackground, in some approaches. Via the thumbnail, a user may review theresults of what was captured and/or processed. If the user isdissatisfied with the result, or the result is otherwise determined tobe unacceptable, e.g. according to one or more predefined qualityassurance metrics, a user interface may facilitate re-capturing and/oraugmenting the originally captured data.

In addition, user interfaces may be provided to enable and/or facilitateuser review of capture and/or processing results, for example at the endof a capture-and-process session. For instance, in one approach a user,upon completion of a capture and/or processing workflow (e.g. videoand/or image data have been captured and at least one processingoperation performed on the data), the user may be presented with anopportunity to review the result of the workflow.

In another approach, user review may be enabled during the video captureoperation. For example, a user initiates the video capture functionalityof a mobile application, and begins capturing video data. As describedabove, the capture operation includes preprocessing such as stabilitydetermination and/or object discovery. In the course of capturing thevideo data, an object in the capture field is detected and an indicationof the discovery is presented to the user (for example the appearance ofa bounding box within the capture field changing color from red togreen). A high-resolution image, thumbnail, etc. is optionally capturedupon discovering the object and determining the existence of a stabilitycondition, and the image may be presented to the user for immediatereview within the video capture user interface. Upon reviewing theimage, thumbnail, etc., the user may indicate the acceptability of thecaptured image, generated thumbnail, etc. If the user indicates theimage, thumbnail, etc. is acceptable, then the video capture userinterface may automatically terminate the capture operation, oroptionally may direct the user to terminate the capture operation. Inthis manner, user review may be utilized to minimize the occurrence ofunnecessary capture and/or processing operations, such as may be causedby a user continuing to perform a capture operation after a suitablehigh-resolution image has been captured and/or processed in a mannerthat satisfies requirements for downstream processing, such as imagequality, image format, etc.

Tracking

In various approaches, the presently disclosed techniques benefit fromthe advantage of real-time (or near-real time) latency. In other words,as a user interacting with a mobile device conducts a capture operation,an analysis, etc. as disclosed herein, the underlying processesconducted to accomplish each operation may be performed in parallel,i.e. for multiple objects simultaneously, and in a near-real timemanner. The computational cost has been reduced to an extent necessaryto provide real-time information regarding object(s) depicted in amobile device viewfinder, and represents a major advantage to the userwhen compared to existing techniques that require discrete capture,analysis, and submission techniques.

As a result, one of the advantageous embodiments of real-time capture,processing, analysis, and etc. is the ability to “track” objectsthroughout the course of performing the presently disclosed techniques.By “tracking” it should be understood that an object within a mobiledevice field of view may be identified and/or analyzed, and theidentification/analysis may remain valid and/or present in a series ofdiscrete frames of image and/or video data because the methodology iscapable of monitoring the position of objects upon detecting thoseobjects, and continuously analyzing the detected objects to provideuseful information.

From the user perspective, tracking typically is embodied in the form ofa bounding border (e.g. box, as described herein) being maintained withrespect to a detected object, even as the mobile device is moved inthree-dimensional space during the capture operation (causing thedetected object to apparently move from the perspective of the mobiledevice's reference point), and/or even as multiple objects are presentin the field of view. Indeed, tracking is capable of monitoring anynumber of objects that may be defined according to characteristics suchas set forth herein.

As will be appreciated by a skilled artisan upon reading the presentdisclosures, any of the raw and/or processed data, such as image data,video data, etc., may be associated with various metadata, may beassociated with other raw or processed data, etc. Moreover, any of thepresently disclosed functionalities may be applied to image capture andprocessing, video capture and processing, etc.

In a preferred approach, for example, tracking comprises one or more of:repositioning or redefining the bounding border to surround theperiphery of the object in each of the frames where the tracked objectis depicted within the viewfinder; and repositioning or redisplaying theindication that the object depicted in the viewfinder exhibits the oneor more defining characteristics.

Optionally, the tracking further comprises receiving real-time feedbackfrom the mobile device. The real-time feedback is based at least inpart, and ideally based entirely, on one or more measurements performedusing mobile device hardware components, for example any one or more of:a camera, an accelerometer, a gyroscope, and a clock.

According to some techniques, the real-time feedback may includestability feedback including an angle of orientation of the mobiledevice being within a predetermined orientation range; and a motionvector of the mobile device having a magnitude less than a predeterminedthreshold.

In another approach, the motion vector of the mobile device isdetermined based on real-time feedback received from the camera, and notdetermined based on feedback from the mobile device accelerometer. Inshort, the tracking techniques are capable of calculating a magnitudeand direction of a velocity with which a camera is being moved throughthree-dimensional space independent of any change in acceleration. As aresult, the device is freed from reliance on an accelerometer todetermine motion vectors (such as would be the case with a constantvelocity).

Yet another advantage conferred by use of video data with the presentinventive techniques is the capacity to generate composite images from aplurality of frames. In one context, this advantage is leveraged asdiscussed below with respect to super-resolution, which may be used toclarify blurred or grainy regions of an otherwise high-quality image,for example.

In the additionally advantageous technique, composite image synthesismay be leveraged to effectively image an object that is otherwise toolarge to capture with sufficient detail for the desired applicationusing image data alone. For example, consider the case of a longdocument such as a receipt or legal form. The document depicts aplethora of informative text, albeit in a relatively small size. Inorder to capture the entire document in a single image, a user wouldhave to distance the document so far from the camera that the quality ofthe informative text would be so degraded that subsequent extraction anduse of the information would be impractical or impossible.

Accordingly, it is an additional aspect of the presently disclosedinventive techniques that along document may be captured using videodata, and the various frames of the video data may be “stitched”together to form a composite image depicting the entire object that wastoo large to fit in a single shot with sufficient clarity. Particularlypreferred are embodiments where the composite image retains the highlevel of specificity and detail otherwise achievable only by zooming inon the object to an extent that capturing the entire object in a singleimage is impossible.

Put another way, in one approach the composite image is characterized bya height and a width. The composite image height is greater than orequal to a height of any single frame of the video data, and thecomposite image width is greater than or equal to a width of any singleframe of the video data. Each of the synthesized frames of the videodata depicts a portion of the object, and the composite image depictsthe entire object.

Moreover still, synthesizing the composite image includes detecting afirst feature (e.g. top border of a page) of the object depicted in theviewfinder; automatically initiating a capture operation in response todetecting the first border of the object capturing one or more ofhigh-resolution image data and low-resolution video data via theautomatically initiated capture operation; detecting a second featuree.g. bottom border of a page) of the object depicted in the viewfinder;capturing one or more of high-resolution image data and low-resolutionvideo data via the automatically initiated capture operation; andautomatically terminating the capture operation in response to detectingthe second feature of the object.

According to the foregoing approach, for example, a user may initiatethe stitching operation by capturing along document using a slow panfrom top to bottom. As discussed in further detail above, windowed OCRmay be particularly advantageous to utilize in combination with astitching-based approach to capture, determine, analyze, etc. textualinformation depicted in a long document or other large object incapableof being captured with a desired level of detail or resolution in asingle image or video frame. For example, in some approaches since auser will capture various partial images from which to form thecomposite image, and this capture process generally (but notnecessarily) involves a relatively smooth, slow panning of the mobiledevice camera with respect to the object being imaged, it will bepossible for the user to simultaneously capture the large object andperform some real-time (or near-real-time) windowed OCR in unison. Thewindowed OCR results may be utilized independently or in conjunctionwith any other defining characteristics of the object to determinecharacteristics of the object. For example, in one approach an objectclassification or pertinent information from the object may bedetermined contemporaneous to capturing the video and/or image data forsubsequent use in generating the composite image.

In various approaches, a suspected object classification could bedetermined based on the defining characteristics of the object, andvalidated using windowed OCR results, e.g. presence of a feature, valueor string known to be present in objects belonging to a particularclassification. Similarly, an object classification could be determinedsolely based on windowed OCR results rather than using definingcharacteristics from the object as described above. Moreover still, insome embodiments both the defining characteristics of the object and thewindowed OCR results may be taken into consideration when determining aclassification of the object.

For example, in one approach an object classification may be determinedindependently based on (1) windowed OCR results and (2) the definingcharacteristics of the object. The independently determined results maybe assigned a relative weight, confidence value, etc., and analyzedfurther in order to make an overall determination with respect to theobject classification.

For instance, various object types may be more readily classifiedaccording to either textual information depicted on the object, or fromdefining characteristics of the object itself. Documents may, forexample, be more readily or accurately classified based on textualinformation that may be determined using a windowed OCR approach, whileobjects depicting an identifying mark such as a logo, emblem, barcode,seal, etc. may be more readily or accurately classified based ondefining characteristics such as object shape, contour, dimensions,color profile, etc. as would be understood by one having ordinary skillin the art upon reading the present descriptions.

Nonetheless, it may be useful to utilize multiple types of informationin determining a classification of a particular object, even if it isknown a priori that the object is more readily or accurately classifiedbased on one specific type of information. In such cases, the form orforms of information that are known to produce facile, accurate,reliable classification of a particular object may be utilized topredict an object classification, and assigned a relative weight (e.g.defining characteristics as defined herein may be utilized and assigneda weight of 0.75 on a 0-to-1 scale).

Similarly, other form s) of information helpful in determining objectclassifications, but perhaps to a lesser degree than the preferredinformation type, may be utilized to predict the object classification,and assigned relatively low weights compared to the preferredinformation type (e.g. textual information determined by windowed OCRmay be used to predict the object classification, and the predictionassigned a weight of 0.25 on a 0-to-1 scale).

Returning now to the concept of composite image synthesis, in order todetermine whether and precisely how two images should be stitched toform the composite image, in one approach the synthesizing comprises:performing at least one homography transformation on two or more of theframes of the video data, and aligning at least portions of the two ormore frames of the video data based at least in part on the homographytransformations.

Several exemplary methods commensurate in scope with the presentdisclosures will now be discussed with particular reference to FIGS. 3and 4. The exemplary embodiments presented below are not to beconsidered limiting on the scope of the instant disclosure, but ratherare provided to illustrate possible implementations of the subjectmatter discussed herein.

Referring now to FIG. 3, a flowchart of a method 300 is shown, accordingto one embodiment. The method 300 may be performed in any suitableenvironment, such as those depicted above in FIGS. 1-2, among others.Moreover, the method 300 may include any number of additional and/oralternative operations aside from those specifically depicted in FIG. 3,in several approaches. The operations of method 300 may be performed inany suitable order that would be comprehended by one having ordinaryskill in the art upon reading this disclosure.

In operation 302, digital video data captured by a mobile device isreceived.

Operations 304-308 may be performed using a processor, which in variousembodiments may be a processor of the mobile device, a processor of aremote device such as a server or another mobile device, a processor ofone or more resources of a cloud computing environment, etc. Operations304-308 may be performed using any combination of such device(s) and/orprocessors thereof, in various embodiments.

In operation 304 a plurality of frames of the digital video data areanalyzed.

In operation 306, one or more frames are identified based on theanalysis, the identified frames satisfying one or more predefinedquality control criteria.

In operation 308, at least one frame satisfying one or more of thepredefined quality control criteria is processed.

In some approaches, predefined quality control criteria may include aminimum illumination level, e.g. an average illumination above a certainthreshold; a maximum illumination level, e.g. an average illuminationbelow a certain threshold; a minimum illumination evenness, e.g. anillumination deviation from some predefined value, from an averageillumination, etc. being below a certain threshold; a minimumresolution; a minimum sharpness, e.g. an amount of blur below a certainthreshold; and a minimum projection, i.e. the impact of projectiveeffects such as angle of camera orientation, fish-bowling, etc. is belowa certain threshold, which may be determined based on metadata collectedduring the capture operation or characteristics of the image.

Quality control criteria may further include, for example, a thresholdvisibility criterion or any other suitable indication of whether thedocument is wholly or partially visible), as may be embodied in athreshold number of expected edges, corners, or other defining featuresare discernable within the viewfinder region and/or are sufficientlywithin the viewfinder region (e.g. first embodiment is a binary yes/no,second embodiment is a further test to see if there is sufficient spacearound each edge of the object, etc. Further still, quality controlcriteria may in various approaches include a presence of glare; and anobject classification.

For example, as a representation of a truly rectangular document may beanalyzed to determine the rectangular “character” of the depicteddocument, which may appear trapezoidal due to imperfect capture angle.Images depicting a document whose appearance deviates too much from“rectangular” may be ignored. Determining whether a shape issubstantially rectangular, such as a shape whose sides correspond toedges of a document, may be accomplished using any known means in theart, and in one embodiment may involve performing one or moretransformations.

In more approaches, the method may additionally and/or alternativelyinclude: determining whether the one or more frames satisfying the oneor more predefined control criteria correspond to a high-resolutionimage stored on the mobile device; processing the high-resolution imageupon determining the one or more frames satisfying the one or morepredefined control criteria correspond to the high-resolution image. Inother words, if a mobile device has a high resolution image of adocument stored in memory, and a video stream captures a relativelylow-resolution but otherwise acceptable frame or frames of the document,it may be preferable to utilize the high-resolution image in subsequentprocessing, but more computationally efficient to capture, analyzeand/or pre-process the relatively low-resolution frame(s) correspondingto the high resolution image.

In various embodiments particularly directed to document processing, thedigital video data comprises a digital representation of a document. Inthis scenario, the method may also include capturing the digital videodata using a camera of the mobile device and detecting the digitalrepresentation of the document.

Some approaches including super-resolution capabilities as describedherein may involve synthesizing at least a portion of two or more framesof the digital video data; and generating a composite image based on thesynthesizing. At least a portion of the composite image is preferablycharacterized by a relatively higher resolution than a resolution of anyof the two or more frames of the digital video data from which thecomposite image was synthesized. The composite image may be utilized toperform document detection (or object discovery in the case of objectsother than documents).

Selective Auto-Capture

Also within the scope of the present disclosure is selectiveauto-capture functionality, which in one embodiment may be implementedin whole or in part as a method, such as method 400, shown in FIG. 4.The method 400 may be performed in any suitable environment, such asthose depicted above in FIGS. 1-2, among others. Moreover, the method400 may include any number of additional and/or alternative operationsaside from those specifically depicted in FIG. 4, in several approaches.The operations of method 400 may be performed in any suitable order thatwould be comprehended by one having ordinary skill in the art uponreading this disclosure.

In operation 402, a mobile application is invoked using a processor of amobile device. The mobile application may be invoked in any suitablemanner, such as by interacting with a user interface of the mobiledevice, issuing a voice command, pressing a button, etc.

In operation 404, a video capture interface of the mobile application isinvoked. The video capture interface may be invoked expressly by a user,for example by interacting with a button or user interface displayed onthe mobile device screen. Alternatively, the video capture interface maybe invoked automatically, either as part of a predetermined routine, inresponse to a precondition being satisfied (such as a prior processcompleting execution), etc., in various approaches.

In operation 406, user input is received via the capture interface. Theuser input may preferably direct the mobile application to invoke acapture operation, for example using a capture component of a mobiledevice.

In operation 408, real-time feedback is requested via the mobile device.The real-time feedback may relate to any relevant portion of videocapture and/or processing, and in one preferred embodiment real-timefeedback relates to invoking the capture operation, such as one or morecharacteristics of data captured via the capture operation, parametersfor performing the capture operation, characteristics of a capturecomponent to be used in performing a capture operation, such as anorientation and/or acceleration of a mobile device (which may bedetermined using integrated hardware components such as a gyroscope, anaccelerometer, etc.), information regarding the result of a captureoperation, suitability of captured data for subsequent processing, etc.

In one particular embodiment, operation 408 includes requestingstability information from one or more hardware components integratedinto the mobile device. The capture interface transparently requestsmobile device acceleration data from an integrated accelerometer inresponse to receiving the user input directing the capture interface toinvoke the capture operation. The capture interface may alsotransparently request mobile device orientation data from an integratedgyroscope in response to receiving the user input directing the captureinterface to invoke the capture operation. Upon receiving the requestedacceleration data and orientation data, the data are compared topredefined stability-threshold criteria previously determined tocorrespond to notability condition, i.e. conditions that typicallyresult in capturing image and/or video data of sufficient quality fordownstream processing. The comparison result may be utilized todetermine whether the stability condition exists, and data may becaptured only upon determining the stability condition exists tomaximize the probability that any image captured via the captureinterface is suitable for the desired downstream processing.

In operation 410, real-time feedback is received. As described infurther detail below, the real-time feedback may be utilized tofacilitate capturing video and/or image data under conditions likely toresult in the video and/or image data being of sufficient quality fordownstream processing. For example, real-time feedback may be utilizedto ensure adequate illumination during capture, to minimize blur, glare,streaking, etc. to ensure the video and/or image data captures anappropriate object or object type, etc.

In operation 412, a capture operation is invoked via the captureinterface upon determining the real-time feedback meets one or morepredetermined criteria.

Real-time feedback is preferably based at least in part on one or moremeasurements performed using one or more integrated hardware componentsof the mobile device. Exemplary integrated hardware components includeone or more of a camera, an accelerometer, a gyroscope, and a clock, butmay include any hardware component integrated into a mobile device.Moreover, the real-time feedback may be in whole or in part anindication that a document is in a field of view of the cameraintegrated into the mobile device.

In a particularly preferred approach, the real-time feedback comprisesstability feedback, such as an angle of orientation of the mobiledevice; an acceleration vector of the mobile device (e.g. a magnitudeand a direction of acceleration per unit time of the mobile device),illumination of a field of view of the camera, illumination of a targetobject in the field of view of the camera, presence of glare in a fieldof view of the camera, etc. as would be understood by one havingordinary skill in the art upon reading the present descriptions. In oneembodiment, illumination may be indicated by and/or derived from aminimum, maximum, average, or other statistical information regardingvideo stream and/or image intensity, brightness, etc., which may beobtained from the camera directly and/or with minimal preprocessingduring the video stream capture, in various approaches. Similarly, glaremay be indicated by one or more regions of the field of view beingoversaturated. Oversaturation may be determined substantially asdisclosed in related U.S. patent application Ser. No. 13/740,123.

Various implementations may utilize predetermined criteria such as theangle of orientation of the mobile device being within a predeterminedorientation range; and the acceleration vector of the mobile devicehaving a magnitude less than a predetermined threshold.

In one approach, invoking the capture operation includes invoking anautofocus operation using a camera of the mobile device; invoking anautoflash operation using the camera; and invoking a data acquisitionoperation using the camera.

In further approaches, data acquisition may include capturing datacomprising one or more of a still image and digital video. The data maybe or comprise a digital representation of a document, or a digitalrepresentation of a plurality of documents.

Still more embodiments within the scope of this disclosure mayadditionally and/or alternatively include determining whether the one ormore predefined criteria are satisfied; and determining whether thecapture operation captured data corresponding to the one or morepredefined criteria being satisfied in response to determining the oneor more predefined criteria are satisfied. For example, one embodimentmay include determining that the mobile device captured data underconditions where the predefined criteria were satisfied, such as aminimum stability, proper angle of orientation, minimum movement in aparticular direction, etc. as would be understood by one having ordinaryskill in the art upon reading the present descriptions.

Exemplary methods within the scope of the instant descriptions mayfurther encompass outputting an indication to the user via a display ofthe mobile device in response to determining the capture operationcaptured data corresponding to the one or more predefined criteria beingsatisfied. The indication preferably indicates the capture operationcaptured data corresponding to the one or more predefined criteria beingsatisfied. In one embodiment, the mobile device may display a boundingborder, box or other overlaying shape around an object depicted in themobile device camera's field of view.

The bounding box may change color upon determining that the captureoperation has completed and captured data under conditions where thepredefined criteria were satisfied. In one approach, the mobile devicedisplays a white bounding box before detecting the object in thecamera's field of view, a yellow bounding box upon detecting the objectbut before completing the capture operation under the desiredconditions, and a green bounding box upon completing the captureoperation under the desired conditions or a red bounding box uponfailing to complete the capture operation under the desired conditions.

In this manner the user may be advantageously informed when to ceaseattempting to capture data via the capture interface of the mobileapplication and/or whether it will be necessary to repeat the captureoperation for any object in the camera's field of view. Of course, inother embodiments where multiple documents are within the camera's fieldof view, the display may display several bounding boxes.

In another embodiment, the one or more bounding boxes may change incolor in response to determining whether one or more of multipledocuments within the camera's field of view have been recently capturedand/or processed. For example, in one approach where a camera's field ofview encompasses several objects such as documents, desirable captureconditions for each object may be achieved at a different point in timeor several different points in time. Accordingly, it may be advantageousto determine when an object in the field of view has been capturedaccording to desirable capture conditions, and cease attempting tocapture that object while continuing to attempt capturing other objectnot yet captured under the desired capture conditions. Similarly, onceall objects have been captured under the desired capture conditions, itmay be useful to provide feedback indicating that all objects depictedin the field of view have been captured according to the desired captureconditions and the capture operation may be ceased.

In still another embodiment, where a camera's field of view encompassesseveral objects such as documents it may be advantageous to exclude orignore objects in the course of the capture operation, for example ifthe object has been previously captured under suitable conditions or ifthe object has been previously processed by and/or according toprocessing intended to be performed using the data currently beingcaptured.

An exemplary use case scenario will now be described, according toseveral embodiments. Those having ordinary skill in the art willappreciate upon reading these descriptions that the exemplary use caseis provided for illustrative purposes, and is not intended to belimiting in any way. Other use cases are fully within the scope of thepresent descriptions, and may include utilizing any combination offeatures disclosed herein in any manner.

According to the exemplary use case scenario, the presently disclosedinventive concepts are embodied in a methodology substantially asrepresented by method 500 as shown in FIG. 5. The method 500 may beperformed in any suitable environment disclosed herein or as would beappreciated by one having ordinary skill in the art upon reading thepresent descriptions.

As shown in FIG. 5, method 500 includes operation 502, where a captureinterface is invoked via a mobile device. The capture interface includesa viewfinder, preferably a rectangular viewfinder defined by a targetingreticle displayed via the mobile device.

Method 500 also includes operation 504, where a plurality of capturedvideo data frames are analyzed to determine (1) whether an objectexhibiting predetermined defining characteristics is wholly or partiallydepicted in the viewfinder region, and (2) whether the object satisfiesquality control criteria. Defining characteristics and quality controlcriteria may include any feature as described herein, preferably thosecharacteristics discussed above and in the related applicationsincorporated by reference with respect to image or objectclassification. Feature vectors represent data particularly suitable foruse as “defining characteristics.”

According to method 500, and depending on whether the object exhibitsthe defining Characteristics and satisfies quality control criteria, oneor more responsive actions are taken in operation 506.

If the aforementioned criteria are met (object detected, qualityacceptable), then an indication of this detection and/or quality statusmay be displayed to the user, e.g. via the device display. Preferably,these indications are displayed in real- or near-real time as the imageanalysis and/or processing are conducted.

On the other hand, if an object is not detected, or does not satisfy thequality control criteria, an indication of the failure(s) may similarlybe displayed via the mobile device.

Further still, one or more images may be captured at a resolution higherthan the resolution of the video data frames (to provide more and/orbetter raw data) and processed or stored. Similarly, those frames ofvideo in which the object was depicted in the viewfinder and satisfiesthe quality control criteria may be archived, flagged, preserved storedto memory, etc. as would be understood by one having ordinary skill inthe art upon reading the present descriptions.

The method 500 may also feature one or more additional and/oralternative operations, in several approaches. For example, in oneapproach the captured image(s) and/or video frame(s) may be furtherprocessed. For example, processing may include classifying object(s)depicted in the image(s) and/or frame(s), extracting data from thoseobject(s), tracking objects depicted in a video stream in real-time,displaying pertinent information to a user, etc.

Moreover, in particularly preferred embodiments, the presently disclosedtechniques are fully capable of simultaneously performing any of thedisclosed processing operations in near-real time as a video stream iscaptured, and furthermore capable of simultaneously performing thecapture, analysis, and/or processing techniques disclosed herein inreal-time or near-real time for multiple objects depicted within asingle video stream (or image). This parallel, real-time functionalityshould be understood to apply equally to the operations discussed abovewith reference to FIG. 5, in various approaches.

In another use case illustration, a user starts a mobile application ontheir mobile device. The user is presented the option to invoke anoptional module such as an “auto-assist” module or a “Mobile CaptureWizard” to assist the user in capturing high quality image data forsubsequent processing. The user may interact with a button displayed onthe mobile device display to invoke the optional “auto-assist” module or“Mobile Capture Wizard,” for example. Alternatively, the module may beprogrammatically invoked or preset by the user, for example in a“settings” interface of the mobile capture application. In this usecase, the user invokes the Mobile Capture Wizard.

Upon invoking the Mobile Capture Wizard, the user is presented with aninterface via the mobile device display, the interface including severalbuttons which permit the user to selectively capture either in a “Photo”mode or a “Video” mode. The user interacts with the “Video” buttonindicating to the mobile application that the user wishes to capture avideo stream. Alternatively, the capture mode may be programmaticallyinvoked or preset by the user, for example in a “settings” interface ofthe mobile capture application.

After indicating the desire to capture a video stream, the user mayinteract with a camera button. In response, the Mobile Capture Wizardqueries the device accelerometer and/or gyroscope to determine deviceorientation, and if the device is not oriented substantially as desired,(e.g. parallel to a flat, horizontal surface such as a desk, the ground,etc. for a document, or parallel to a vertical surface such as a poster,automobile surface depicting a Vehicle Identification Number, etc.),user feedback may be presented to the user via the mobile devicedisplay, e.g. in the form of a transparent screen, the feedbackindicating improper device orientation. The user feedback may overlaythe capture interface viewport so that the user cannot capture an imageuntil proper device orientation is achieved. Proper device orientationmay be programmatically determined and/or preset by a user to includeany device orientation (e.g. as an angle) in a settings interface.

As the user moves the orientation of the device to a “desiredorientation,” the mobile application periodically queries the deviceaccelerometer and/or gyroscope to determine the actual orientation ofthe device. During this operation an on-screen user feedback isdisplayed indicating to the user how the orientation of the deviceshould be adjusted for optimal capture orientation.

Once the orientation falls within a predetermined tolerance range foraspect ratio correction and normalization, the transparent overlayscreen disappears and the mobile application begins analyzing, inreal-time, video data gathered from the mobile device camera to detectpage boundaries.

Upon detecting page boundaries, the mobile device optionally againperiodically checks for device stability, and upon determining that thedevice is sufficiently stable according to predetermined tolerancethresholds, additional user feedback is provided indicating suitableconditions exist for capturing the detected page. For example, userfeedback may be provided in the form of a yellow bounding box displayedaround the detected page boundaries.

Whether or not the optional second orientation and/or stabilitydetection operation is performed, upon determining that the device isstable, properly oriented, and a page has been detected, user feedbackis displayed via the mobile device display to indicate conditionssuitable for capturing high-quality image data exist. The mobileapplication then proceeds automatically to capturing the image of thedocument depicted in the capture interface viewport.

The mobile application capture then forces an auto-focus operation usingthe device camera, and captures the video frame containing the detectedpage, which may be a high resolution version, if available, of theoriginal video frame containing the detected page. Alternatively, one ormore video frames, including the relatively low-resolution originalvideo frame in which the page was detected may be captured. The mobileapplication displays the captured image in real-time, either directly inthe capture interface or in another interface displayed on the mobiledevice display after completing the high-resolution capture operation.

Full image processing which may include any or all of the imageprocessing operations disclosed in the related U.S. patent applicationsreferenced above) is initiated by the mobile application as abackground, asynchronous thread. In addition, a quality control processmodule is simultaneously initiated and an on screen indication is givenas feedback regarding document illumination and/or blur. Once theasynchronous background processing is complete, the displayed image isrefreshed (e.g. top to bottom on the mobile device display) with theprocessed image. Notably, the image processing may be performed using aprocessor of the mobile device, a processor of a remote device such as aserver, or any combination thereof.

The mobile application may either automatically save the original andprocessed image, or prompt a user for instructions regarding whether tosave the original and/or processed image, and save the images indicatedby the user accordingly. The data may be synchronized with a host cloudor on-premises system for storage, further processing and/or subsequentre-use.

In various embodiments, saving the image, whether the original image,processed image, or any variations thereof, may include saving a filecombined with any image-related metadata, such as classificationresults, extraction results, or any environmental metadata such asgeo-position tagging, date/time stamping, etc. all within one singlefile (e.g. a printable document format (PDF) e-form). This type of saveoperation may be optionally invoked by the user in real-time or in asettings interface. Alternatively, the image data may be saved as-iswithout being associated with metadata.

Various embodiments may additionally and/or alternatively includedisplaying a capture result preview via a display of the mobile deviceand receiving user input in response to the capture preview. Moreover,in at least one embodiment the preview depicts at least one objectcaptured via the capture operation.

The inventive concepts disclosed herein have been presented by way ofexample to illustrate the myriad features thereof in a plurality ofillustrative scenarios, embodiments, and/or implementations. It shouldbe appreciated that the concepts generally disclosed are to beconsidered as modular, and may be implemented in any combination,permutation, or synthesis thereof. In addition, any modification,alteration, or equivalent of the presently disclosed features,functions, and concepts that would be appreciated by a person havingordinary skill in the art upon reading the instant descriptions shouldalso be considered within the scope of this disclosure.

For example, in myriad illustrative approaches, a method, a systemconfigured to execute logic and perform a method, and/or a computerprogram product comprising computer readable instructions configured tocause a processor to perform a method may include any one or more of thefollowing features. Similarly, various embodiments may exclude some orall of the features set forth below. In general, the following featuresmay be combined in any suitable manner that would be appreciated by onehaving ordinary skill in the art upon reading the present descriptions.

Again, a method, system, and/or computer program product may include anycombination of: invoking an image capture interface via a mobile device,the capture interface comprising a viewfinder represented on a displayof the mobile device; and analyzing a plurality of frames of video datacaptured via the capture interface. The analyzing may includedetermining: Whether an object exhibiting one or more definingcharacteristics is depicted within the viewfinder; and if so, whetherthe object depicted within the viewfinder satisfies one or morepredetermined quality control criteria. In response to determining aframe fails one or more of the predetermined quality control criteria,the method/system/computer program may include displaying an indicationof the failure on the mobile device display. The failure indicationpreferably identifies the one or more quality control criteria notsatisfied by the frame(s), and optionally includes abounding bordersubstantially surrounding a periphery of the object within theviewfinder. In response to determining the object depicted within theviewfinder satisfies the one or more predetermined quality controlcriteria, the method/system/computer program may include one or more ofdisplaying an indication that the object depicted in the viewfinderexhibits the one or more defining characteristics; automaticallycapturing an image of the object, wherein the image is characterized bya resolution higher than a resolution of the video data; andautomatically storing to a memory one or more of the frames in which theobject satisfying the predetermined quality control criteria is depictedin the viewfinder. Preferably, the success indicator specificallyidentifies object classification, and optionally includes a boundingborder substantially surrounding a periphery of the object within theviewfinder. In some approaches, the object comprises a document havingone or more pages, or multiple documents each having one or more pages.The method/system/computer program may include processing at least oneof the automatically captured image and the automatically storedframe(s) at least in part using a processor of the mobile device. Theprocessing includes tracking the object depicted within the viewfinderin real-time or near-real-time; classifying the object depicted withinthe viewfinder; and/or extracting data from the object depicted in theviewfinder. Processing is optionally performed with respect to at leasttwo of the multiple documents or multiple pages, when present; e.g.tracking, classifying and/or extracting may be performed simultaneouslywith respect to one another and/or multiple documents or pages of asingle document. The simultaneous processing may preferably be conductedacross multiple frames of the video data: Classifying may morespecifically include: determining one or more defining characteristicsof the object depicted within the viewfinder; and comparing the one ormore determined defining characteristics to defining characteristics ofeach of a plurality of object classifications. In some situations, themethod/system/computer program includes either: determining an objectclassification based at least in part on the comparison; or determiningthe object does not correspond to any of the plurality of objectclassifications based at least in part on the comparison. In response todetermining the object does not correspond to any of the plurality ofobject classifications the method/system/computer program includes:requesting user input relating to the object:receiving the user input;defining a new object classification based at least in part on the userinput; and assigning the object to the new object classification.Similarly, tracking may include one or more of: repositioning orredefining the bounding border to surround the periphery of the objectin each of the frames where the object is depicted within theviewfinder; repositioning or redisplaying the indication that the objectdepicted in the viewfinder exhibits the one or more definingcharacteristics; and receiving real-time feedback from the mobiledevice, the real-time feedback being based at least in part on one ormore measurements performed using one or more mobile device componentsselected from: a camera, an accelerometer, a gyroscope, and a clock.Preferably, the real-time feedback includes stability feedback includingan angle of orientation of the mobile device being within apredetermined orientation range; and a motion vector of the mobiledevice having a magnitude less than a predetermined threshold. Themotion vector, in some approaches is determined based on real-timefeedback received from the camera, and is not determined based onfeedback from an accelerometer. In some approaches, particularly thoseinvolving tracking or processing of long/large objects, themethod/system/computer program also includes synthesizing at least aportion of two or more frames of the video data; and generating acomposite image based on the synthesis. The composite image isoptionally characterized by a height and a width, and the compositeimage height is greater than or equal to a height of any single frame ofthe video data, and/or the composite image width is similarly greaterthan or equal to a width of any single frame of the video data. Each ofthe synthesized frames of the video data ideally depicts a portion ofthe object, while the composite image depicts an entirety of the object.Synthesizing the composite image from the various video data framespreferably includes: detecting a first feature of the object depicted inthe viewfinder; automatically initiating a capture operation in responseto detecting the first border of the object; capturing one or more ofhigh-resolution image data and low-resolution video data via theautomatically initialed capture operation; detecting a second feature ofthe object depicted in the viewfinder; capturing one or more ofhigh-resolution image data and low-resolution video data via theautomatically initiated capture operation; and automatically terminatingthe capture operation in response to detecting the second feature of theobject. Synthesizing may also, or instead, include performing at leastone homography transformation on two or more of the frames of the videodata, and aligning at least portions of the two or more frames of thevideo data based at least in part on the homography transformations. Theobject imaged using the aforementioned synthesizing techniques to form acomposite is preferably a document characterized by at least onedimension thereof being too large to encompass the entire documentwithin the viewfinder, and simultaneously preserve a desired minimumresolution of the document, e.g. a resolution sufficient to resolvetextual information depicted in the document. Moreover still, at least aportion of the composite image may be characterized by a higherresolution than a resolution of any of the two or more frames of thedigital video data. This result may be accomplished, in someembodiments, using a “super-resolution” technique as described herein,and may further include detecting the object in the composite imagebased at least in part on the portion of the composite imagecharacterized by the higher resolution. Determining whether the one ormore frames satisfying the one or more predefined control criteriacorrespond to a high-resolution image stored on the mobile device isanother useful feature; and may be used to retrieve previously capturedand stored high-resolution images, as well as process the storedhigh-resolution image(s). It may be useful in some scenarios toassociate metadata with stored image and/or video data. In preferredapproaches, predetermined quality control criteria may include any oneor more of: a minimum illumination level; a maximum illumination level;a minimum illumination evenness; a minimum resolution; a minimumsharpness; a minimum projection angle; a maximum projection angle; athreshold visibility criterion; a presence of glare; and an objectclassification.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of an embodiment of the presentinvention should not be limited by any of the above-described exemplaryembodiments, but should be defined only in accordance with the followingclaims and their equivalents.

What is claimed is:
 1. A computer-implemented method, comprising:receiving image data via a mobile device, the image data comprising aplurality of frames; and generating a composite image based on at leasttwo of the plurality of frames.
 2. The method as recited in claim 1,wherein the at least two of the plurality of frames are eachcharacterized by a reduced resolution; and wherein the composite imageis characterized by a higher resolution.
 3. The method as recited inclaim 2, wherein the reduced resolution is a resolution in a range fromabout 256 pixels by about 256 pixels to about 480 pixels by about 270pixels; and wherein the higher resolution is at least about 1000 pixelsby about 1000 pixels.
 4. The method as recited in claim 1, wherein theat least two of the plurality of frames are each characterized by ablurred region; and wherein the composite image is characterized by anabsence of the blurred region.
 5. The method as recited in claim 4,wherein a location of the blurred region in each of the at least two ofthe plurality of frames is the same; and wherein a location of theabsence of the blurred region in the composite image is the same as thelocation of the blurred region in each of the at least two of theplurality of frames.
 6. The method as recited in claim 1, wherein eachof the at least two of the plurality of frames comprises binary imagedata, and the method comprising one or more of comparing, merging, andextracting data from one or more of the composite image and the at leasttwo of the plurality of frames.
 7. The method as recited in claim 6,wherein the data comprises text characters.
 8. The method as recited inclaim 1, comprising capturing the image data, wherein the image datacomprise video data.
 9. The method as recited in claim 1, wherein theimage data depict a plurality of objects, wherein each of the pluralityof objects is depicted in a different set of one or more of theplurality of frames; wherein the composite image depicts at least one ofthe plurality of objects; wherein one or more of the at least two framesare selected from one of the different sets of one or more frames; andwherein one or more other of the at least two frames are selected fromanother of the different sets of one or more frames.
 10. The method asrecited in claim 1, wherein each of the plurality of frames isCharacterized by a different resolution ranging from about 25 pixels byabout 25 pixels to about 256 pixels by about 256 pixels.
 11. The methodas recited in claim 10, comprising determining whether one or more ofthe frames depict an object exhibiting one or more definingcharacteristics.
 12. The method as recited in claim 11, comprising, inresponse to determining one or more of the frames depict the object,setting a minimum capture resolution for capturing other objectsexhibiting the one or more defining characteristics.
 13. The method asrecited in claim 12, comprising receiving user feedback confirming ornegating whether the one or more frames depict the object, wherein theminimum capture resolution is set in further response to determining theuser feedback confirms the one or more frames depict the object.
 14. Themethod as recited in claim 12, wherein the minimum capture resolution isat least as high as a lowest solution of the one or more framesdetermined to depict the object.
 15. The method as recited in claim 10,comprising determining whether one or more of the frame(s) determined todepict the object satisfy one or more predetermined quality controlcriteria.
 16. The method as recited in claim 15, comprising, in responseto determining one or more of the frames depict the object, setting aminimum capture resolution for capturing other objects exhibiting theone or more defining characteristics.
 17. The method as recited in claim16, wherein the minimum capture resolution is at least as high as alowest resolution of the one or more frames determined to depict theobject.
 18. A computer program product comprising: a computer readablestorage medium having program code embodied therewith, the program codereadable/executable by a processor to: receive image data via a mobiledevice, the image data comprising a plurality of frames; and generate acomposite image based on at least two of the plurality of frames. 19.The computer program product as recited in claim 18, wherein the atleast two of the plurality of frames are each characterized by a reducedresolution; and wherein the composite image is characterized by a higherresolution.
 20. The computer program product as recited in claim 18,wherein the at least two of the plurality of frames are eachcharacterized by a blurred region; and wherein the composite image ischaracterized by absence of the blurred region.